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Program Description 1 

MyTeachingPartner-Secondary (MTP-S) is a professional develop- 
ment program that aims to increase student learning and development 
through improved teacher-student interactions. Through the program, 
middle and high school teachers access a video library featuring 
examples of high-quality interactions and receive individualized, 
web-based coaching approximately twice per month during the school 
year. MTP-S uses the secondary school version of the Classroom 
Assessment Scoring System®-Secondary (CLASS-S) to define and 
observe effective teaching practices. 

Research 2 

The What Works Clearinghouse (WWC) identified one study of MTP-S 
that both falls within the scope of the Teacher Training, Evaluation, 
and Compensation topic area and meets WWC group design 
standards. This one study meets WWC group design standards with 
reservations. The study included 1 ,267 middle and high school 
students in 12 schools in Virginia. 3 

The WWC considers the extent of evidence for MTP-S on the aca- 
demic achievement of middle and high school students to be small 
for one outcome domain— general achievement. There were no studies 
that meet WWC design standards in the five other student outcome 

domains and six teacher outcome domains, so this intervention report does not report on the effectiveness of 
MTP-S for those domains. (See the Effectiveness Summary on p. 4 for more details of effectiveness by domain.) 

Effectiveness 45 

MTP-S was found to have no discernible effects on general achievement for middle and high school students. 
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review protocol, version 3.1. 


Table 1. Summary of findings 




Improvement index (percentile points) 




Outcome domain 

Rating of 
effectiveness 

Average 

Range 

Number of 
studies 

Number of 
students 

Extent of 
evidence 

General achievement 

No discernible 
effects 

+2 

na 

1 

1,267 

Small 


na = not applicable 
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Program Information 

Background 

The Center for Advanced Study of Teaching and Learning (CASTL) developed MTP-S. Robert Pianta, founding 
director of CASTL, and Bridget Hamre, associate director of CASTL, founded Teachstone to scale implementation 
of MTP-S and other CASTL-developed tools. 

CASTL’s address is 350 Old Ivy Way, Suite 202, Charlottesville, VA 22903. Web: www.mtpsecondary.net and 
http://curry.virginia.edu/research/centers/castl/project/mtp-Secondary. Telephone: (434) 243-2406. Email: 
chris.hafen@virginia.edu. 

Teachstone’s address is 105 Monticello Avenue, Suite 201 , Charlottesville, VA 22902. Web: www.teachstone.com. 
Telephone: (866) 998-8352. Fax: (434) 293-4338. Email: contact@teachstone.com. 

Program details 

MTP-S has two components: 

• A video library of annotated examples of best practice. The online video library contains more than 400 
one- to two-minute video clips of teachers interacting with students. A description accompanying each 
video clip explains the behavior shown and why the interaction is effective. The videos, which are organized 
by CLASS-S dimensions, feature examples of a variety of instructional activities in different content areas 
with students from diverse socioeconomic backgrounds. The CLASS-S dimensions are organized into three 
domains: (1) Emotional Support, which includes positive climate, negative climate, teacher sensitivity, and 
regard for adolescent perspectives; (2) Classroom Organization, which includes behavior management, pro- 
ductivity, and instructional learning formats; and (3) Instructional Support, which includes content understand- 
ing, analysis and problem solving, and quality of feedback. 

• Web-based individualized coaching. Each teacher participating in the MTP-S coaching program receives 
web-based support from a trained consultant throughout the school year. Twice a month, teachers submit 
videotaped sessions of their own instruction to a consultant, who identifies segments of the session for review 
and provides feedback linked to effective practices from the video library. Teachers review the identified 
segments for their own behaviors and for student reactions and then confer with the consultant, who recom- 
mends strategies to enhance teacher-student interactions. 

MTP-S is currently available in two formats: a direct teacher support model and a coach support model. In the 
direct teacher support model, Teachstone delivers the MTP-S coaching program directly to teachers. The coach 
support model uses a “train-the-trainer” approach, in which Teachstone trains and supports school and district 
staff in serving as MTP-S coaches, with the level of Teachstone support decreasing over multiple years of imple- 
mentation. The study reviewed in this report used the direct teacher support model. 


Cost 

MTP-S can be purchased by schools, districts, training and technical assistance centers, and institutes of higher 
education. The direct teacher support model currently costs $5,000 per teacher. The cost of the coach support 
model varies from $8,500 per coach in the start-up phase with biweekly support, to $4,000 per coach in the final 
phase of semiannual support; an additional fee of $325 per teacher in the coach support model covers teacher 
materials and equipment. Each coach can support up to 18 teachers under the coach support model. Specific 
pricing may vary based on factors such as the number of teachers and coaches. 
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Research Summary 

The WWC identified three eligible studies that investigated the effects 
of MTP-S on the academic achievement of middle and high school 
students. An additional three studies were identified but do not meet 
WWC eligibility criteria for review in this topic area. Citations for all six 
studies are in the References section, which begins on p. 5. 

The WWC reviewed three eligible studies against group design stan- 
dards. One study (Allen, Pianta, Gregory, Mikami, & Lun, 2011) is a randomized controlled trial that meets WWC 
group design standards with reservations. The study is summarized in this report. Two studies do not meet 
WWC group design standards. 

Summary of studies meeting WWC group design standards without reservations 

No studies of MTP-S met WWC group design standards without reservations. 

Summary of study meeting WWC group design standards with reservations 

Allen et al. (2011) examined the effectiveness of MTP-S using a randomized controlled trial conducted in 12 middle 
and high schools in Virginia. Participating teachers in each school were grouped by the subject (i.e., math/science, 
language arts/social studies) they would teach in the evaluation. Within each subject in each school, participating 
teachers were randomly assigned to either the MTP-S intervention or “business-as-usual” professional develop- 
ment. Teachers were then asked to select their “focal class” for inclusion in the evaluation— the class they expected 
would be most academically challenging and had end-of-year achievement tests. Although initial selection of 
focal classes occurred before teachers learned their research condition, some teachers later changed their focal 
classes. 6 Parental consent and student assent to participate in the evaluation also occurred after randomization. 
Because the teachers’ ultimate selection of focal classes, parental consent, and student assent could have been 
affected by knowledge of the teacher’s research condition, the study was determined by the WWC to have a com- 
promised random assignment process. The study demonstrated baseline equivalence of the analytic samples and, 
therefore, meets WWC group design standards with reservations. 

Student achievement was assessed using end-of-year scores from Commonwealth of Virginia Standards of Learning 
(SOL) tests taken in core subjects, with the SOL score on the most comparable course in the previous school year 
used as a pretest. Because the authors report a single SOL test score without distinguishing academic achieve- 
ment in specific subject areas, the outcome falls in the general achievement domain. The analytic sample included 
76 teachers (39 MTP-S and 37 comparison) and 1 ,267 students (606 MTP-S and 661 comparison) in the interven- 
tion year. 7 The authors also reported supplemental achievement findings from the end of the following school year, 
during which participating teachers taught a different set of students, and teachers in the intervention condition no 
longer received MTP-S coaching. 


Table 2. Scope of reviewed research 


Grades 

6-12 

Delivery method 

Whole class 

Program type 

Teacher level 
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Effectiveness Summary 

The WWC review of MTP-S for the Teacher Training, Evaluation, and Compensation topic area includes both 
student and teacher outcomes. The review encompasses student outcomes in six domains: English language arts 
achievement, mathematics achievement, science achievement, social studies achievement, general achievement, 
and student progression. The review includes teacher outcomes in six domains: teacher instruction, teacher atten- 
dance, student growth scores, teacher retention at the school, teacher retention in the school district, and teacher 
retention in the profession. The one study of MTP-S that meets WWC group design standards reported findings 
in one of the six domains for student outcomes: general achievement. The findings below present the authors’ 
estimates and WWC-calculated estimates of the size and statistical significance of the effects of MTP-S on middle 
and high school students. An additional comparison is presented as a supplemental finding. This supplemental 
finding does not factor into the intervention’s rating of effectiveness. For a more detailed description of the rating of 
effectiveness and extent of evidence criteria, see the WWC Rating Criteria on p. 12. 

Summary of effectiveness for the general achievement domain 

One study that meets WWC group design standards with reservations reported findings in the general achievement 
domain. 

Allen et al. (2011) examined one outcome in the general achievement domain: an end-of-year score from SOL tests 
taken in core subjects. The authors reported, and the WWC confirmed, that the difference between the MTP-S 
group and the comparison group at the end of the intervention year was not statistically significant. According to 
WWC criteria, the effect size was not large enough to be considered substantively important (i.e., an effect size of 
at least 0.25). The WWC characterizes these study findings as an indeterminate effect. 

Thus, for the general achievement domain, one study showed indeterminate effects. This results in a rating of no 
discernible effects, with a small extent of evidence. 


Table 3. Rating of effectiveness and extent of evidence for the general achievement domain 


Rating of effectiveness 

Criteria met 

No discernible effects 

No affirmative evidence of effects. 

In the one study that reported findings, the estimated impact of the intervention on outcomes in the general 
achievement domain was neither statistically significant nor large enough to be substantively important. 

Extent of evidence 

Criteria met 

Small 

One study that included 1 ,267 students in 12 schools reported evidence of effectiveness in the general 
achievement domain. 
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ED535637.pdf The study is ineligible for review because it does not use an eligible design. 
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Appendix A: Research details for Allen et al., 2011 s 

Allen, J. P., Pianta, R. C., Gregory, A., Mikami, A. Y., & Lun, J. (2011). An interaction-based approach to 
enhancing secondary school instruction and student achievement. Science, 333(6045), 1034-1037. 


Table A. Summary of findings 


Meets WWC group design standards with reservations 

Outcome domain 

Sample size 

Study findings 

Average improvement index 

(percentile points) Statistically significant 

General achievement 

76 teachers/1,267 students 

+2 No 


Setting The study was conducted in 12 middle and high schools in Virginia. 

Study sample The study recruited secondary school teachers who expected to be the primary instructor of 
a course that had end-of-course standardized achievement exams. Participating teachers in 
each school were grouped by the subject (i.e., math/science, language arts/social studies) 
they would teach in the evaluation. Within each subject in each school, participating teach- 
ers were randomly assigned to either the MTP-S intervention or “business-as-usual” profes- 
sional development. Teachers were then asked to select their “focal class” for inclusion in the 
evaluation— the class that they expected to be the most academically challenging and that 
had end-of-year achievement tests. Although initial selection of focal classes occurred before 
teachers learned their research condition, some teachers later changed their focal classes. 
Written parental consent and student assent were obtained for participating students in the 
focal classes after randomization. Because the teachers’ ultimate selection of focal classes, 
parental consent, and student assent could have been affected by knowledge of the teacher’s 
research condition, the study was determined by the WWC to have a compromised random 
assignment process. 

The study included 78 teachers (40 MTP-S and 38 comparison) from 12 schools at the begin- 
ning of the intervention year. 9 The analytic sample included 1 ,267 students (606 MTP-S and 
661 comparison) taught by 76 teachers (39 MTP-S and 37 comparison) in the intervention 
year. The mean grade of the students was grade 8. Forty-six percent of students were female, 
and 31 % had families with incomes at less than 200% of the poverty line. The racial/ethnic 
demographics were as follows: 71 % were White, 23% were Black, 5% were Hispanic, 1 % 
were Asian, and less than 1 % were of other race/ethnicity. 10 

The authors also reported achievement findings from the end of the following school year, 
during which participating teachers taught a different set of students, and teachers in the 
intervention condition no longer received MTP-S coaching. The analytic sample for the post- 
intervention year included a new set of 970 students (419 MTP-S and 551 comparison) who 
had not participated in the study during the intervention year, but were taught by 61 teachers 
(27 MTP-S and 34 comparison) who had participated in the study during the intervention year. 
Per the Teacher Training, Evaluation, and Compensation review protocol, the post-intervention 
year findings are presented as supplemental findings in Appendix D. The supplemental find- 
ings do not factor into the intervention’s rating of effectiveness. 
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Intervention 

group 


Comparison 

group 


Outcomes and 
measurement 


Support for 
implementation 


Intervention group teachers attended an initial workshop, where consultants defined the 
MTP-S principles and described the dimensions of high-quality teacher-student interactions 
from the CLASS-S. Approximately twice a month throughout the school year, each teacher 
submitted videotaped sessions from his or her focal class to a consultant, who identified brief 
segments of the session for review and discussion. The teacher reviewed the identified seg- 
ments for his or her own behaviors and for student reactions and answered questions from 
the consultant about the connection between the behaviors and the reactions. The teacher 
then conferred with the consultant in a 20- to 30-minute discussion by phone in which the 
consultant recommended strategies to enhance teacher-student interactions. Consultants 
also directed teachers to view annotated video exemplars of high-quality teaching available 
on the MTP-S website. The year of coaching was followed by a brief “booster” workshop. The 
intervention involved a total of about 20 hours of in-service training over 13 months. The mean 
years of teaching experience was 7.6. Sixty-nine percent of MTP-S teachers were female, and 
59% held a master’s or higher level degree. 

Comparison group teachers received regular in-service training. They videotaped six class- 
room segments throughout the school year at times that coincided with MTP-S teacher vid- 
eotapings, but they did not receive feedback on the videotapes. Comparison teachers did not 
have access to the MTP-S library of video exemplars. The mean years of teaching experience 
was 10.1. Fifty-seven percent of comparison teachers were female, and 70% held a master’s 
or higher level degree. 

An outcome in the general achievement domain was reported. Student achievement was mea- 
sured using end-of-year scores from tests taken in core subjects, with the score on the most 
comparable course in the previous school year used as a pretest. Because the authors report 
a single test score without distinguishing academic achievement in specific subject areas, 
the outcome falls in the general achievement domain. For a more detailed description of this 
outcome measure, see Appendix B. 

Prior to the beginning of the school year, both MTP-S and comparison group teachers partici- 
pated in a 3-hour workshop in which they received instruction in procedures for videotaping 
classes and submitting the videotapes (as well as procedures for study-related tasks, such as 
obtaining student assent/parent consent and collecting self-reported data). Consultants for the 
MTP-S teachers were master teachers trained in using the CLASS-S system. Two consultants 
led an initial workshop for intervention teachers that outlined MTP-S principles and discussed 
the CLASS-S dimensions of high-quality teacher-student interactions. 
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Appendix B: Outcome measure for the general achievement domain 


General achievement 


Commonwealth of Virginia Standards of These standardized tests comprise the accreditation testing program for Virginia. The tests are administered to 

Learning (SOL) tests students at the end of core subject courses. Each test consists of between 45 and 63 multiple-choice questions 

and is standardized on a 200 to 600 point scale (as cited in Allen et at, 2011). 
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Appendix C: Findings included in the rating for the general achievement domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Allen etal., 2011 a 

Commonwealth of Virginia 
Standards of Learning (SOL) 
tests 

Middle 
and high 
school 

76 teachers/ 
1,267 students 

468.9 

(74.4) 

465.6 

(78.0) 

3.32 

0.04 

+2 

>.05 


Not 

Domain average for general achievement (Allen etal., 2011) 0.04 +2 statistically 

significant 


Domain average for general achievement across all studies 0.04 +2 na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are 
given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in 
an average individual’s percentile rank that can be expected if the individual is given the intervention. The statistical significance of the study’s domain average was determined by 
the WWC. Some statistics may not sum as expected due to rounding, na = not applicable. 

a For Allen et al. (201 1 ), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was reported 
in the original study. The WWC calculated the intervention group mean by adding the impact of the intervention (the estimated coefficient on the intervention group indicator from a 
hierarchical linear model) to the unadjusted comparison group posttest mean. This study is characterized as having an indeterminate effect because the estimated effect is neither 
statistically significant nor substantively important. For more information, please refer to the WWC Procedures and Standards Handbook (version 3.0), p. 26. 
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Appendix D: Post-intervention year findings for the general achievement domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Allen etal., 2011 a 

Commonwealth of Virginia 

Middle and 

61 teachers/ 

498.0 

482.2 

15.77 

0.23 

+9 

<,05 

Standards of Learning 

high school 

970 students 

(72.0) 

(65.9) 






(SOL) tests 

Table Notes: The supplemental findings presented in this table are additional findings from studies in this report that do not factor into the determination of the intervention rating. 
For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors the compari- 
son group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are given the 
intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in an average 
individual's percentile rank that can be expected if the individual is given the intervention. Some statistics may not sum as expected due to rounding. 

a For Allen et al. (201 1 ), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was reported 
in the original study. The WWC calculated the intervention group mean by adding the impact of the intervention (the estimated coefficient on the intervention group indicator from a 
hierarchical linear model) to the unadjusted comparison group posttest mean. 
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Endnotes 

1 The descriptive information for this program was obtained from publicly available sources: the program’s websites (www.mtpsecondary. 
net and http://curry.virginia.edu/research/centers/castl, downloaded March 201 4) and the research literature (Allen et al., 201 1 ; Pianta, 

201 1). The WWC requests developers review the program description sections for accuracy from their perspective. The program descrip- 
tion was provided to the developer in March 2014, and the WWC incorporated feedback from the developer. Further verification of the 
accuracy of the descriptive information for this program is beyond the scope of this review. 

2 The literature search reflects documents publicly available by July 2014. The studies in this report were reviewed using the Standards 
from the WWC Procedures and Standards Handbook (version 3.0), along with those described in the Teacher Training, Evaluation, 
and Compensation review protocol (version 3.1). The evidence presented in this report is based on available research. Findings and 
conclusions may change as new research becomes available. 

3 The study sample included 78 teachers in 12 schools, but the analytic sample for the finding on which the evidence rating is based 
included only 76 teachers. The authors did not report the number of schools included in the analytic sample; however, there is no 
indication that any schools were dropped. 

4 For criteria used in the determination of the rating of effectiveness and extent of evidence, see the WWC Rating Criteria on p. 12. These 
improvement index numbers show the average and range of individual-level improvement indices for all findings across the studies. 

5 The following domains were not examined by studies that meet WWC design standards: English language arts achievement, math- 
ematics achievement, science achievement, social studies achievement, student progression, teacher instruction, teacher attendance, 
student growth scores, teacher retention at the school, teacher retention in the school district, and teacher retention in the profession. 

6 Information about the timing of the selection of focal classes relative to randomization was provided by the study authors at the 
WWC’s request. 

7 The student analytic sample sizes by condition were provided by the study authors at the WWC’s request. 

8 The WWC identified four additional sources related to Allen et al. (2011). These studies do not contribute unique information to 
Appendix A and are not listed here. 

9 The teacher sample sizes by condition were provided by the study authors at the WWC’s request. 

10 The WWC calculated these sample characteristics, which are the student-weighted averages of the characteristics that the authors 
reported separately for the intervention and comparison groups. Intervention and comparison group percentages differed by less than 
3 percentage points for each characteristic; none of the differences was statistically significant. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2015, June). Teacher 
Training, Evaluation, and Compensation intervention report: MyTeachingPartner-Secondary. Retrieved from 
http://whatworks.ed.gov 
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WWC Rating Criteria 

Criteria used to determine the rating of a study 

Study rating 

Criteria 

Meets WWC group design 
standards without reservations 

A study that provides strong evidence for an intervention’s effectiveness, such as a well-implemented RCT. 

Meets WWC group design 

A study that provides weaker evidence for an intervention's effectiveness, such as a QED or an RCT with high 

standards with reservations 

attrition that has established equivalence of the analytic samples. 

Criteria used to determine the rating of effectiveness for an intervention 

Rating of effectiveness 

Criteria 

Positive effects 

Two or more studies show statistically significant positive effects, at least one of which met WWC group design 
standards for a strong design, AND 

No studies show statistically significant or substantively important negative effects. 

Potentially positive effects 

At least one study shows a statistically significant or substantively important positive effect, AND 

No studies show a statistically significant or substantively important negative effect AND fewer or the same number 

of studies show indeterminate effects than show statistically significant or substantively important positive effects. 

Mixed effects 

At least one study shows a statistically significant or substantively important positive effect AND at least one study 
shows a statistically significant or substantively important negative effect, but no more such studies than the number 
showing a statistically significant or substantively important positive effect, OR 

At least one study shows a statistically significant or substantively important effect AND more studies show an 
indeterminate effect than show a statistically significant or substantively important effect. 

Potentially negative effects 

One study shows a statistically significant or substantively important negative effect and no studies show 
a statistically significant or substantively important positive effect, OR 

Two or more studies show statistically significant or substantively important negative effects, at least one study 
shows a statistically significant or substantively important positive effect, and more studies show statistically 
significant or substantively important negative effects than show statistically significant or substantively important 
positive effects. 

Negative effects 

Two or more studies show statistically significant negative effects, at least one of which met WWC group design 
standards for a strong design, AND 

No studies show statistically significant or substantively important positive effects. 

No discernible effects 

None of the studies shows a statistically significant or substantively important effect, either positive or negative. 

Criteria used to determine the extent of evidence for an intervention 

Extent of evidence 

Criteria 

Medium to large 

The domain includes more than one study, AND 
The domain includes more than one school, AND 

The domain findings are based on a total sample size of at least 350 students, OR, assuming 25 students in a class, 
a total of at least 14 classrooms across studies. 

Small 

The domain includes only one study, OR 
The domain includes only one school, OR 

The domain findings are based on a total sample size of fewer than 350 students, AND, assuming 25 students 
in a class, a total of fewer than 14 classrooms across studies. 
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Glossary of Terms 

Attrition 

Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Extent of evidence 

Improvement index 

Intervention 
Intervention report 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Rating of effectiveness 


Single-case design 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 

A demonstration that the analytic sample groups are similar on observed characteristics 
defined in the review area protocol. 

An indication of how much evidence supports the findings. The criteria for the extent 
of evidence levels are given in the WWC Rating Criteria on p. 12. 

Along a percentile distribution of individuals, the improvement index represents the gain 
or loss of the average individual due to the intervention. As the average individual starts at 
the 50th percentile, the measure ranges from -50 to +50. 

An educational program, product, practice, or policy aimed at improving student outcomes. 

A summary of the findings of the highest-quality research on a given program, product, 
practice, or policy in education. The WWC searches for all research studies on an interven- 
tion, reviews each against design standards, and summarizes the findings of those that 
meet WWC design standards. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which study participants are 
assigned to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which eligible study participants are 
randomly assigned to intervention and comparison groups. 

The WWC rates the effects of an intervention in each domain based on the quality of the 
research design and the magnitude, statistical significance, and consistency in findings. The 
criteria for the ratings of effectiveness are given in the WWC Rating Criteria on p. 12. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 
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Standard deviation The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 

Statistical significance Statistical significance is the probability that the difference between groups is a result of 

chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < .05). 


Substantively important a substantively important finding is one that has an effect size of 0.25 or greater, regardless 

of statistical significance. 


Systematic review a review of existing literature on a topic that is identified and reviewed using explicit meth- 
ods. A WWC systematic review has five steps: 1) developing a review protocol; 2) searching 
the literature; 3) reviewing studies, including screening studies for eligibility, reviewing the 
methodological quality of each study, and reporting on high-quality studies and their find- 
ings; 4) combining findings within and across studies; and, 5) summarizing the review. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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Intervention 

Report 



Practice 

Guide 



Quick 

Review 


Single Study 
Review 



An intervention report summarizes the findings of high-quality research on a given program, practice, or policy in 
education. The WWC searches for all research studies on an intervention, reviews each against evidence standards, 
and summarizes the findings of those that meet standards. 
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